{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "title:generic,gcp",
    "tags": []
   },
   "source": [
    "# Getting Started with Feature Store"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "In this notebook, you learn how to use Vertex AI Feature Store for training and prediction with Vertex AI.\n",
    "\n",
    "- Create a Vertex AI Featurestore resource.\n",
    "    - Create EntityType resources for the Featurestore resource.\n",
    "    - Create Feature resources for each EntityType resource.\n",
    "- Import feature values (entity data items) into the Featurestore resource.\n",
    "- Perform online serving from the Featurestore resource.\n",
    "- Perform batch serving from the Featurestore resource.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "overview:mlops"
   },
   "source": [
    "## Overview\n",
    "\n",
    "This notebook demonstrates how to use Vertex AI for end-to-end MLOps on Google Cloud in production. This notebook uses Vertex AI Feature Store, a Google Cloud ML service."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dataset:movies,lbn,avro",
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Dataset\n",
    "\n",
    "The dataset used for this notebook is the Movie Recommendations. The version of the dataset you will use in this notebook is stored in a public Cloud Storage bucket, in Avro format.\n",
    "\n",
    "The dataset predicts whether a persons will watch a movie.\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in the [student lab notebook](../labs/7_get_started_with_feature_store.ipynb) -- try to complete that notebook first before reviewing this solution notebook.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "install_mlops"
   },
   "source": [
    "## Installations\n",
    "\n",
    "Install *one time* the packages for executing the MLOps notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "install_mlops"
   },
   "outputs": [],
   "source": [
    "ONCE_ONLY = False\n",
    "if ONCE_ONLY:\n",
    "    ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
    "    ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
    "    ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
    "    ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
    "    ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
    "    ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
    "    ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
    "    ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
    "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
    "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
    "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
    "    ! pip3 install --upgrade kfp $USER_FLAG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "restart"
   },
   "source": [
    "### Restart the kernel\n",
    "\n",
    "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "restart"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "project_id"
   },
   "source": [
    "#### Set your project ID\n",
    "\n",
    "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "set_project_id"
   },
   "outputs": [],
   "source": [
    "PROJECT_ID = \"<Your-Project-ID>\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "autoset_project_id"
   },
   "outputs": [],
   "source": [
    "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
    "    # Get your GCP project id from gcloud\n",
    "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "    PROJECT_ID = shell_output[0]\n",
    "    print(\"Project ID:\", PROJECT_ID)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "set_gcloud_project_id"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n"
     ]
    }
   ],
   "source": [
    "! gcloud config set project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "region"
   },
   "source": [
    "#### Region\n",
    "\n",
    "You can also change the `REGION` variable, which is used for operations\n",
    "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
    "\n",
    "- Americas: `us-central1`\n",
    "- Europe: `europe-west4`\n",
    "- Asia Pacific: `asia-east1`\n",
    "\n",
    "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
    "\n",
    "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "region"
   },
   "outputs": [],
   "source": [
    "REGION = \"us-central1\"  # @param {type: \"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "timestamp"
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "setup_vars"
   },
   "source": [
    "### Set up variables\n",
    "\n",
    "Next, set up some variables used throughout the notebook.\n",
    "### Import libraries and define constants"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "import_aip"
   },
   "source": [
    "#### Import Vertex SDK\n",
    "\n",
    "Import the Vertex SDK into your Python environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "import_aip"
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "from google.cloud.aiplatform import gapic as aip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "import_bq"
   },
   "source": [
    "#### Import BigQuery\n",
    "\n",
    "Import the BigQuery package into your Python environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "import_bq"
   },
   "outputs": [],
   "source": [
    "from google.cloud import bigquery"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "init_bq"
   },
   "source": [
    "### Create BigQuery client\n",
    "\n",
    "Create the BigQuery client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "init_bq"
   },
   "outputs": [],
   "source": [
    "bqclient = bigquery.Client()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aip_constants:fs"
   },
   "source": [
    "#### Vertex AI constants\n",
    "\n",
    "Setup up the following constants for Vertex AI:\n",
    "\n",
    "- `API_ENDPOINT`: The Vertex AI API service endpoint for `FeatureStore` services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "aip_constants:fs"
   },
   "outputs": [],
   "source": [
    "# API service endpoint\n",
    "API_ENDPOINT = \"{}-aiplatform.googleapis.com\".format(REGION)\n",
    "\n",
    "# Vertex location root path for your dataset, model and endpoint resources\n",
    "PARENT = \"projects/\" + PROJECT_ID + \"/locations/\" + REGION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "clients:featurestore"
   },
   "source": [
    "## Set up clients\n",
    "\n",
    "The Vertex SDK works as a client/server model. On your side (the Python script) you will create a client that sends requests and receives responses from the Vertex AI server.\n",
    "\n",
    "You will use different clients in this notebook for different steps in the workflow. So set them all up upfront.\n",
    "\n",
    "- Feature Store Service for creating a feature store.\n",
    "- Feature Store Serving Service for serving from a feature store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "clients:featurestore"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('feature_store', <google.cloud.aiplatform_v1.services.featurestore_service.client.FeaturestoreServiceClient object at 0x7fa6dbbf8110>)\n",
      "('feature_store_serving', <google.cloud.aiplatform_v1.services.featurestore_online_serving_service.client.FeaturestoreOnlineServingServiceClient object at 0x7fa6dbbf80d0>)\n"
     ]
    }
   ],
   "source": [
    "# client options same for all services\n",
    "client_options = {\"api_endpoint\": API_ENDPOINT}\n",
    "\n",
    "\n",
    "def create_feature_store_client():\n",
    "    client = aip.FeaturestoreServiceClient(client_options=client_options)\n",
    "    return client\n",
    "\n",
    "\n",
    "def create_feature_store_serving_client():\n",
    "    client = aip.FeaturestoreOnlineServingServiceClient(client_options=client_options)\n",
    "    return client\n",
    "\n",
    "\n",
    "clients = {}\n",
    "clients[\"feature_store\"] = create_feature_store_client()\n",
    "clients[\"feature_store_serving\"] = create_feature_store_serving_client()\n",
    "\n",
    "for client in clients.items():\n",
    "    print(client)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_intro"
   },
   "source": [
    "## Introduction to Vertex AI Feature Store\n",
    "\n",
    "Let's assume you have a recommendation model that predicts a coupon to print on the back of a cash register receipt. Now, if that model was trained only on single transaction instances (what was bought and how much), then (in the past) you used an Apriori algorithm.\n",
    "\n",
    "But now we have historical data on the customer (say it's indexed by credit card number). Like total purchases to date, average purchase per transaction, frequency of purchase by product category, etc. We use this \"enriched data\" to train a recommender system.\n",
    "\n",
    "Now it's time to do a live prediction. You get a transaction from the cash register, but all it has is the credit card number and this transaction. It does not have the enriched data the model needs. During serving, the credit card number is used as an index to Feature Store to get the enriched data needed for the model.\n",
    "\n",
    "Next problem. Let's say the enriched data the model was trained on was timestamp June 1. This transaction is June 15. Assume that the user has made other transactions between June 1 and 15, and the enriched data has been continuously updated in Feature Store. But the model was trained on June 1st data. FeatureStore knows the version number and serves the June 1 version to the model (not the current June 15); otherwise, if you used June 15 data you have training-serving skew.\n",
    "\n",
    "Next problem, data drift. Things change, suddenly one day everybody is buying toilet paper! There is a significant change in the distribution of the current stored enriched data from the distribution that the deployed model was trained on. FeatureStore can detect changes/thresholds in distribution changes and trigger a notification for retraining the model.\n",
    "\n",
    "Learn more about [Vertex AI Feature Store API](https://cloud.google.com/vertex-ai/docs/featurestore)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_datamodel:movies"
   },
   "source": [
    "## Vertex AI Feature Store data model\n",
    "\n",
    "Vertex AI Feature Store organizes data with the following 3 important hierarchical concepts:\n",
    "\n",
    "        Featurestore -> EntityType -> Feature\n",
    "\n",
    "- `Featurestore`: the place to store your features\n",
    "- `EntityType`: under a `Featurestore`, an `EntityType` describes an object to be modeled, real one or virtual one.\n",
    "- `Feature`: under an `EntityType`, a `Feature` describes an attribute of the `EntityType`\n",
    "\n",
    "Learn more about [Vertex AI Feature Store data model](https://cloud.google.com/vertex-ai/docs/featurestore/concepts).\n",
    "\n",
    "In the movie prediction dataset, you create a `Featurestore` resource called movies. This `Featurestore` resource has 2 entity types:\n",
    "- `users`: The entity type has the `age`, `gender`, and `like genres` features.\n",
    "- `movies`: The entity type has the `genres` and `average rating` features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_create"
   },
   "source": [
    "## Create a `Featurestore` resource\n",
    "\n",
    "First, you create a `Featurestore` for the dataset using the `create_featurestore()` method, with the following parameters:\n",
    "\n",
    "- `parent`: The base portion of the fully qualified resource identifier (projects/<project>/location/<location>)\n",
    "- `featurestore_id`: The name of the feature store.\n",
    "- `featurestore`: Configuration settings for the feature store.\n",
    "    - `online_serving_config`: Configuration settings for online serving from the feature store.\n",
    "\n",
    "Note, this is a long-running-operation (LRO), so you do a `response.result()` to block on the operation completing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "featurestore_create"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name: \"projects/541313880005/locations/us-central1/featurestores/movies\""
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from google.cloud.aiplatform_v1.types import featurestore, featurestore_service\n",
    "\n",
    "# Represents featurestore resource path.\n",
    "FEATURESTORE_NAME = \"movies\"\n",
    "\n",
    "response = clients[\"feature_store\"].create_featurestore(\n",
    "    featurestore_service.CreateFeaturestoreRequest(\n",
    "        parent=PARENT,\n",
    "        featurestore_id=FEATURESTORE_NAME,\n",
    "        featurestore=featurestore.Featurestore(\n",
    "            online_serving_config=featurestore.Featurestore.OnlineServingConfig(\n",
    "                fixed_node_count=1\n",
    "            )\n",
    "        ),\n",
    "    )\n",
    ")\n",
    "\n",
    "response.result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_list"
   },
   "source": [
    "### List your `Featurestore` resources\n",
    "\n",
    "You can get a list of all your `Featurestore` resources in your project using the `list_featurestores()` method, with the following parameters:\n",
    "\n",
    "- `parent`: The base portion of the fully qualified resource identifier (projects/<project>/location/<location>)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "featurestore_list"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: \"projects/541313880005/locations/us-central1/featurestores/movies\"\n",
      "create_time {\n",
      "  seconds: 1642789770\n",
      "  nanos: 454781000\n",
      "}\n",
      "update_time {\n",
      "  seconds: 1642789770\n",
      "  nanos: 511273000\n",
      "}\n",
      "etag: \"AMEw9yOOytF7BEsKftzEKuN3RksKUmu_MIZZ1DtyxoaCuNej86UtwhHcq-XhrAj56Yds\"\n",
      "online_serving_config {\n",
      "  fixed_node_count: 1\n",
      "}\n",
      "state: STABLE\n",
      "\n"
     ]
    }
   ],
   "source": [
    "featurestores = clients[\"feature_store\"].list_featurestores(parent=PARENT)\n",
    "\n",
    "for featurestore in featurestores:\n",
    "    print(featurestore)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_get"
   },
   "source": [
    "### Get a `Featurestore` resource\n",
    "\n",
    "You can get a specifed `Featurestore` resource in your project using the `get_featurestore()` method, with the following parameters:\n",
    "\n",
    "- `name`: The fully qualified resource identifier for the `Featurestore` resource."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "featurestore_get"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "projects/qwiklabs-gcp-04-ca29274815c5/locations/us-central1/featurestores/movies\n",
      "name: \"projects/541313880005/locations/us-central1/featurestores/movies\"\n",
      "create_time {\n",
      "  seconds: 1642789770\n",
      "  nanos: 454781000\n",
      "}\n",
      "update_time {\n",
      "  seconds: 1642789770\n",
      "  nanos: 511273000\n",
      "}\n",
      "etag: \"AMEw9yOPnMqlwgeVJl8ujTW2Jm8dNdlgURLUknAWusGEww4lTKdOXyqCqM9OEtFlXQ6D\"\n",
      "online_serving_config {\n",
      "  fixed_node_count: 1\n",
      "}\n",
      "state: STABLE\n",
      "\n"
     ]
    }
   ],
   "source": [
    "resource_name = clients[\"feature_store\"].featurestore_path(\n",
    "    PROJECT_ID, REGION, FEATURESTORE_NAME\n",
    ")\n",
    "print(resource_name)\n",
    "\n",
    "featurestore = clients[\"feature_store\"].get_featurestore(name=resource_name)\n",
    "print(featurestore)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_create:entity_type"
   },
   "source": [
    "## Create entity types for your `Featurestore` resource\n",
    "\n",
    "Next, you create the `EntityType` resources for your `Featurestore` resource using the `create_entity_type()` method, with the following parameters:\n",
    "\n",
    "- `parent`: The fully qualified resource identifier for the `Featurestore` resource.\n",
    "- `entity_type_id`: The name of the `EntityType` resource.\n",
    "- `entity_type`: Configuration settings for the `EntityType` resource."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "featurestore_create:entity_type"
   },
   "outputs": [],
   "source": [
    "from google.cloud.aiplatform_v1.types import entity_type\n",
    "\n",
    "for name, description in [(\"users\", \"Users descrip\"), (\"movies\", \"Movies descrip\")]:\n",
    "    response = clients[\"feature_store\"].create_entity_type(\n",
    "        featurestore_service.CreateEntityTypeRequest(\n",
    "            parent=resource_name,\n",
    "            entity_type_id=name,\n",
    "            entity_type=entity_type.EntityType(\n",
    "                description=description,\n",
    "            ),\n",
    "        )\n",
    "    )\n",
    "\n",
    "    response.result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_create:feature"
   },
   "source": [
    "### Add `Feature` resources for your `EntityType` resources\n",
    "\n",
    "Next, you create the `Feature` resources for each of the `EntityType` resources in your `Featurestore` resource using the `create_feature()` method, with the following parameters:\n",
    "\n",
    "- `parent`: The fully qualified resource identifier for the `EntityType` resource.\n",
    "- `feature_id`: The name of the `Feature` resource.\n",
    "- `feature`: The configuration settings for the `Feature` resource."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "featurestore_create:feature,movies"
   },
   "outputs": [],
   "source": [
    "from google.cloud.aiplatform_v1.types import feature\n",
    "\n",
    "\n",
    "def create_features(featurestore_name, entity_name, features):\n",
    "    parent = clients[\"feature_store\"].entity_type_path(\n",
    "        PROJECT_ID, REGION, featurestore_name, entity_name\n",
    "    )\n",
    "    for name, descrip, dtype in features:\n",
    "\n",
    "# TODO 1: Create the Feature resources for each of the EntityType resources in your Featurestore resource using the create_feature() method.\n",
    "        response = clients[\"feature_store\"].create_feature(\n",
    "            parent=parent,\n",
    "            feature=feature.Feature(value_type=dtype, description=descrip),\n",
    "            feature_id=name,\n",
    "        )\n",
    "\n",
    "        response.result()\n",
    "\n",
    "\n",
    "create_features(\n",
    "    FEATURESTORE_NAME,\n",
    "    \"users\",\n",
    "    [\n",
    "        (\"age\", \"Age descrip\", feature.Feature.ValueType.INT64),\n",
    "        (\"gender\", \"Gender descrip\", feature.Feature.ValueType.STRING),\n",
    "        (\"liked_genres\", \"Genres descrip\", feature.Feature.ValueType.STRING_ARRAY),\n",
    "    ],\n",
    ")\n",
    "\n",
    "create_features(\n",
    "    FEATURESTORE_NAME,\n",
    "    \"movies\",\n",
    "    [\n",
    "        (\"title\", \"Title descrip\", feature.Feature.ValueType.STRING),\n",
    "        (\"genres\", \"Genres descrip\", feature.Feature.ValueType.STRING),\n",
    "        (\"average_rating\", \"Ave descrip\", feature.Feature.ValueType.DOUBLE),\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_search"
   },
   "source": [
    "### Search all `Feature` resources in your `Featurestore` resources\n",
    "\n",
    "You can get a list of all `Feature` resources in your `Featurestore` resources using the method `search_features()`, with the following parameters:\n",
    "\n",
    "- `location`: The base portion of the fully qualified resource identifier (projects/<project>/location/<location>)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "featurestore_search"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SearchFeaturesPager<features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/average_rating\"\n",
      "  description: \"Ave descrip\"\n",
      "  value_type: DOUBLE\n",
      "  create_time {\n",
      "    seconds: 1642789884\n",
      "    nanos: 687682000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789884\n",
      "    nanos: 687682000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/genres\"\n",
      "  description: \"Genres descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789883\n",
      "    nanos: 969453000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789883\n",
      "    nanos: 969453000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/title\"\n",
      "  description: \"Title descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789881\n",
      "    nanos: 631430000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789881\n",
      "    nanos: 631430000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/age\"\n",
      "  description: \"Age descrip\"\n",
      "  value_type: INT64\n",
      "  create_time {\n",
      "    seconds: 1642789877\n",
      "    nanos: 22930000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789877\n",
      "    nanos: 22930000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/gender\"\n",
      "  description: \"Gender descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789878\n",
      "    nanos: 662477000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789878\n",
      "    nanos: 662477000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/liked_genres\"\n",
      "  description: \"Genres descrip\"\n",
      "  value_type: STRING_ARRAY\n",
      "  create_time {\n",
      "    seconds: 1642789880\n",
      "    nanos: 757672000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789880\n",
      "    nanos: 757672000\n",
      "  }\n",
      "}\n",
      ">\n",
      "SearchFeaturesPager<features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/average_rating\"\n",
      "  description: \"Ave descrip\"\n",
      "  value_type: DOUBLE\n",
      "  create_time {\n",
      "    seconds: 1642789884\n",
      "    nanos: 687682000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789884\n",
      "    nanos: 687682000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/genres\"\n",
      "  description: \"Genres descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789883\n",
      "    nanos: 969453000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789883\n",
      "    nanos: 969453000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/title\"\n",
      "  description: \"Title descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789881\n",
      "    nanos: 631430000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789881\n",
      "    nanos: 631430000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/age\"\n",
      "  description: \"Age descrip\"\n",
      "  value_type: INT64\n",
      "  create_time {\n",
      "    seconds: 1642789877\n",
      "    nanos: 22930000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789877\n",
      "    nanos: 22930000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/gender\"\n",
      "  description: \"Gender descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789878\n",
      "    nanos: 662477000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789878\n",
      "    nanos: 662477000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/liked_genres\"\n",
      "  description: \"Genres descrip\"\n",
      "  value_type: STRING_ARRAY\n",
      "  create_time {\n",
      "    seconds: 1642789880\n",
      "    nanos: 757672000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789880\n",
      "    nanos: 757672000\n",
      "  }\n",
      "}\n",
      ">\n",
      "SearchFeaturesPager<features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/average_rating\"\n",
      "  description: \"Ave descrip\"\n",
      "  value_type: DOUBLE\n",
      "  create_time {\n",
      "    seconds: 1642789884\n",
      "    nanos: 687682000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789884\n",
      "    nanos: 687682000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/genres\"\n",
      "  description: \"Genres descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789883\n",
      "    nanos: 969453000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789883\n",
      "    nanos: 969453000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/title\"\n",
      "  description: \"Title descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789881\n",
      "    nanos: 631430000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789881\n",
      "    nanos: 631430000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/age\"\n",
      "  description: \"Age descrip\"\n",
      "  value_type: INT64\n",
      "  create_time {\n",
      "    seconds: 1642789877\n",
      "    nanos: 22930000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789877\n",
      "    nanos: 22930000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/gender\"\n",
      "  description: \"Gender descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789878\n",
      "    nanos: 662477000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789878\n",
      "    nanos: 662477000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/liked_genres\"\n",
      "  description: \"Genres descrip\"\n",
      "  value_type: STRING_ARRAY\n",
      "  create_time {\n",
      "    seconds: 1642789880\n",
      "    nanos: 757672000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789880\n",
      "    nanos: 757672000\n",
      "  }\n",
      "}\n",
      ">\n",
      "SearchFeaturesPager<features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/average_rating\"\n",
      "  description: \"Ave descrip\"\n",
      "  value_type: DOUBLE\n",
      "  create_time {\n",
      "    seconds: 1642789884\n",
      "    nanos: 687682000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789884\n",
      "    nanos: 687682000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/genres\"\n",
      "  description: \"Genres descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789883\n",
      "    nanos: 969453000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789883\n",
      "    nanos: 969453000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/title\"\n",
      "  description: \"Title descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789881\n",
      "    nanos: 631430000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789881\n",
      "    nanos: 631430000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/age\"\n",
      "  description: \"Age descrip\"\n",
      "  value_type: INT64\n",
      "  create_time {\n",
      "    seconds: 1642789877\n",
      "    nanos: 22930000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789877\n",
      "    nanos: 22930000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/gender\"\n",
      "  description: \"Gender descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789878\n",
      "    nanos: 662477000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789878\n",
      "    nanos: 662477000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/liked_genres\"\n",
      "  description: \"Genres descrip\"\n",
      "  value_type: STRING_ARRAY\n",
      "  create_time {\n",
      "    seconds: 1642789880\n",
      "    nanos: 757672000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789880\n",
      "    nanos: 757672000\n",
      "  }\n",
      "}\n",
      ">\n",
      "SearchFeaturesPager<features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/average_rating\"\n",
      "  description: \"Ave descrip\"\n",
      "  value_type: DOUBLE\n",
      "  create_time {\n",
      "    seconds: 1642789884\n",
      "    nanos: 687682000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789884\n",
      "    nanos: 687682000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/genres\"\n",
      "  description: \"Genres descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789883\n",
      "    nanos: 969453000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789883\n",
      "    nanos: 969453000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/title\"\n",
      "  description: \"Title descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789881\n",
      "    nanos: 631430000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789881\n",
      "    nanos: 631430000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/age\"\n",
      "  description: \"Age descrip\"\n",
      "  value_type: INT64\n",
      "  create_time {\n",
      "    seconds: 1642789877\n",
      "    nanos: 22930000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789877\n",
      "    nanos: 22930000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/gender\"\n",
      "  description: \"Gender descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789878\n",
      "    nanos: 662477000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789878\n",
      "    nanos: 662477000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/liked_genres\"\n",
      "  description: \"Genres descrip\"\n",
      "  value_type: STRING_ARRAY\n",
      "  create_time {\n",
      "    seconds: 1642789880\n",
      "    nanos: 757672000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789880\n",
      "    nanos: 757672000\n",
      "  }\n",
      "}\n",
      ">\n",
      "SearchFeaturesPager<features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/average_rating\"\n",
      "  description: \"Ave descrip\"\n",
      "  value_type: DOUBLE\n",
      "  create_time {\n",
      "    seconds: 1642789884\n",
      "    nanos: 687682000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789884\n",
      "    nanos: 687682000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/genres\"\n",
      "  description: \"Genres descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789883\n",
      "    nanos: 969453000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789883\n",
      "    nanos: 969453000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/title\"\n",
      "  description: \"Title descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789881\n",
      "    nanos: 631430000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789881\n",
      "    nanos: 631430000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/age\"\n",
      "  description: \"Age descrip\"\n",
      "  value_type: INT64\n",
      "  create_time {\n",
      "    seconds: 1642789877\n",
      "    nanos: 22930000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789877\n",
      "    nanos: 22930000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/gender\"\n",
      "  description: \"Gender descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789878\n",
      "    nanos: 662477000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789878\n",
      "    nanos: 662477000\n",
      "  }\n",
      "}\n",
      "features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users/features/liked_genres\"\n",
      "  description: \"Genres descrip\"\n",
      "  value_type: STRING_ARRAY\n",
      "  create_time {\n",
      "    seconds: 1642789880\n",
      "    nanos: 757672000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789880\n",
      "    nanos: 757672000\n",
      "  }\n",
      "}\n",
      ">\n"
     ]
    }
   ],
   "source": [
    "features = clients[\"feature_store\"].search_features(location=PARENT)\n",
    "\n",
    "for feature in features:\n",
    "    print(features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_search:query"
   },
   "source": [
    "### Search `Feature` resources using a query filter\n",
    "\n",
    "You can narrow your search of `Feature` resources by specifying a `query` filter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "featurestore_search:query"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "By Name\n",
      "SearchFeaturesPager<features {\n",
      "  name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/title\"\n",
      "  description: \"Title descrip\"\n",
      "  value_type: STRING\n",
      "  create_time {\n",
      "    seconds: 1642789881\n",
      "    nanos: 631430000\n",
      "  }\n",
      "  update_time {\n",
      "    seconds: 1642789881\n",
      "    nanos: 631430000\n",
      "  }\n",
      "}\n",
      ">\n",
      "By Data Type\n",
      "name: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/movies/features/average_rating\"\n",
      "description: \"Ave descrip\"\n",
      "value_type: DOUBLE\n",
      "create_time {\n",
      "  seconds: 1642789884\n",
      "  nanos: 687682000\n",
      "}\n",
      "update_time {\n",
      "  seconds: 1642789884\n",
      "  nanos: 687682000\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Search by name\n",
    "features = clients[\"feature_store\"].search_features(\n",
    "    featurestore_service.SearchFeaturesRequest(\n",
    "        location=PARENT, query=\"feature_id:title\"\n",
    "    )\n",
    ")\n",
    "\n",
    "print(\"By Name\")\n",
    "for feature in features:\n",
    "    print(features)\n",
    "\n",
    "# Search by data type\n",
    "features = clients[\"feature_store\"].search_features(\n",
    "    featurestore_service.SearchFeaturesRequest(\n",
    "        location=PARENT, query=\"value_type=DOUBLE\"\n",
    "    )\n",
    ")\n",
    "\n",
    "print(\"By Data Type\")\n",
    "for feature in features:\n",
    "    print(feature)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "import_file:movies,lbn,avro"
   },
   "outputs": [],
   "source": [
    "IMPORT_FILE = (\n",
    "    \"gs://cloud-samples-data/vertex-ai/feature-store/datasets/movie_prediction.csv\"\n",
    ")\n",
    "FS_ENTITIES = {\n",
    "    \"users\": \"gs://cloud-samples-data/vertex-ai/feature-store/datasets/users.avro\",\n",
    "    \"movies\": \"gs://cloud-samples-data-us-central1/vertex-ai/feature-store/datasets/movies.avro\",\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_import:movies,avro"
   },
   "source": [
    "## Import the feature data into your `Featurestore` resource\n",
    "\n",
    "Next, you import the feature data for your `Featurestore` resource. Once imported, you can use these feature values for online and offline (batch) serving.\n",
    "\n",
    "### Data layout\n",
    "\n",
    "Each imported `EntityType` resource data must have an ID; also, each `EntityType` resource data item can optionally have a timestamp, sepecifying when the feature values were generated.\n",
    "\n",
    "When importing, specify the following in your request:\n",
    "\n",
    "- Data source format: BigQuery Table/Avro/CSV\n",
    "- Data source URL\n",
    "- Destination: featurestore/entity types/features to be imported\n",
    "\n",
    "The feature values for the movies dataset are in Avro format. The Avro schemas are as follows:\n",
    "\n",
    "**Users entity**:\n",
    "\n",
    "```\n",
    "schema = {\n",
    "  \"type\": \"record\",\n",
    "  \"name\": \"User\",\n",
    "  \"fields\": [\n",
    "      {\n",
    "       \"name\":\"user_id\",\n",
    "       \"type\":[\"null\",\"string\"]\n",
    "      },\n",
    "      {\n",
    "       \"name\":\"age\",\n",
    "       \"type\":[\"null\",\"long\"]\n",
    "      },\n",
    "      {\n",
    "       \"name\":\"gender\",\n",
    "       \"type\":[\"null\",\"string\"]\n",
    "      },\n",
    "      {\n",
    "       \"name\":\"liked_genres\",\n",
    "       \"type\":{\"type\":\"array\",\"items\":\"string\"}\n",
    "      },\n",
    "      {\n",
    "       \"name\":\"update_time\",\n",
    "       \"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-micros\"}]\n",
    "      },\n",
    "  ]\n",
    " }\n",
    " ```\n",
    "\n",
    "**Movies entity**:\n",
    "\n",
    "```\n",
    "schema = {\n",
    " \"type\": \"record\",\n",
    " \"name\": \"Movie\",\n",
    " \"fields\": [\n",
    "     {\n",
    "      \"name\":\"movie_id\",\n",
    "      \"type\":[\"null\",\"string\"]\n",
    "     },\n",
    "     {\n",
    "      \"name\":\"average_rating\",\n",
    "      \"type\":[\"null\",\"double\"]\n",
    "     },\n",
    "     {\n",
    "      \"name\":\"title\",\n",
    "      \"type\":[\"null\",\"string\"]\n",
    "     },\n",
    "     {\n",
    "      \"name\":\"genres\",\n",
    "      \"type\":[\"null\",\"string\"]\n",
    "     },\n",
    "     {\n",
    "      \"name\":\"update_time\",\n",
    "      \"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-micros\"}]\n",
    "     },\n",
    " ]\n",
    "}\n",
    "```\n",
    "\n",
    "### Importing the feature values\n",
    "\n",
    "You import the feature values for the `EntityType` resources using the `import_feature_values()` method, with the following parameters:\n",
    "\n",
    "- `entity_type`: The fully qualified resource identifier for the `EntityType` resource.\n",
    "- The location of the feature values, one of:\n",
    "    `avro_source`: The Cloud Storage location of the feature values in Avro format.\n",
    "    `csv_source`: The Cloud Storage location of the feature values in Avro format.\n",
    "    `bigquery_source`: The BigQuery table for the feature values.\n",
    "- `entity_id_field`: The source column for the unique ID for each entity data item.\n",
    "- `feature_specs`: The source colums for the features to import into the `EntityType` resource.\n",
    "- `feature_time_field`: The source column for the timestamp of each entity data item.\n",
    "- `worker_count`: The number of parallel workers to read in and update the feature values in the `EntityType` resource.\n",
    "**Note**: This can take around 10-15 minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "featurestore_import:movies,avro"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "imported_entity_count: 4\n",
       "imported_feature_value_count: 12"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from google.cloud.aiplatform_v1.types import io as io\n",
    "\n",
    "request = featurestore_service.ImportFeatureValuesRequest(\n",
    "    entity_type=clients[\"feature_store\"].entity_type_path(\n",
    "        PROJECT_ID, REGION, FEATURESTORE_NAME, \"users\"\n",
    "    ),\n",
    "    avro_source=io.AvroSource(\n",
    "        # Source\n",
    "        gcs_source=io.GcsSource(uris=[FS_ENTITIES[\"users\"]])\n",
    "    ),\n",
    "    entity_id_field=\"user_id\",\n",
    "    feature_specs=[\n",
    "        # Features\n",
    "        featurestore_service.ImportFeatureValuesRequest.FeatureSpec(id=\"age\"),\n",
    "        featurestore_service.ImportFeatureValuesRequest.FeatureSpec(id=\"gender\"),\n",
    "        featurestore_service.ImportFeatureValuesRequest.FeatureSpec(id=\"liked_genres\"),\n",
    "    ],\n",
    "    feature_time_field=\"update_time\",\n",
    "    worker_count=1,\n",
    ")\n",
    "\n",
    "response = clients[\"feature_store\"].import_feature_values(request)\n",
    "response.result()\n",
    "\n",
    "request = featurestore_service.ImportFeatureValuesRequest(\n",
    "# TODO 2: Import the resource identifier for the EntityType resource.\n",
    "    entity_type=clients[\"feature_store\"].entity_type_path(\n",
    "        PROJECT_ID, REGION, FEATURESTORE_NAME, \"movies\"\n",
    "    ),\n",
    "    avro_source=io.AvroSource(gcs_source=io.GcsSource(uris=[FS_ENTITIES[\"movies\"]])),\n",
    "    entity_id_field=\"movie_id\",\n",
    "    feature_specs=[\n",
    "        featurestore_service.ImportFeatureValuesRequest.FeatureSpec(id=\"title\"),\n",
    "        featurestore_service.ImportFeatureValuesRequest.FeatureSpec(id=\"genres\"),\n",
    "        featurestore_service.ImportFeatureValuesRequest.FeatureSpec(\n",
    "            id=\"average_rating\"\n",
    "        ),\n",
    "    ],\n",
    "    feature_time_field=\"update_time\",\n",
    "    worker_count=1,\n",
    ")\n",
    "\n",
    "response = clients[\"feature_store\"].import_feature_values(request)\n",
    "response.result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_serving"
   },
   "source": [
    "## Vertex AI Feature Store serving\n",
    "\n",
    "The Vertex AI Feature Store service provides the following two services for serving features from a `Featurestore` resource:\n",
    "\n",
    "- Online serving - low-latency serving of small batches of features (prediction).\n",
    "\n",
    "- Batch serving - high-throughput serving of large batches of features (training and prediction)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "featurestore_serving:online,single"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "header {\n",
      "  entity_type: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users\"\n",
      "  feature_descriptors {\n",
      "    id: \"age\"\n",
      "  }\n",
      "  feature_descriptors {\n",
      "    id: \"gender\"\n",
      "  }\n",
      "  feature_descriptors {\n",
      "    id: \"liked_genres\"\n",
      "  }\n",
      "}\n",
      "entity_view {\n",
      "  entity_id: \"alice\"\n",
      "  data {\n",
      "    value {\n",
      "      int64_value: 55\n",
      "      metadata {\n",
      "        generate_time {\n",
      "          seconds: 1629493102\n",
      "          nanos: 261000000\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  data {\n",
      "    value {\n",
      "      string_value: \"Female\"\n",
      "      metadata {\n",
      "        generate_time {\n",
      "          seconds: 1629493102\n",
      "          nanos: 261000000\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  data {\n",
      "    value {\n",
      "      string_array_value {\n",
      "        values: \"Drama\"\n",
      "      }\n",
      "      metadata {\n",
      "        generate_time {\n",
      "          seconds: 1629493102\n",
      "          nanos: 261000000\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from google.cloud.aiplatform_v1.types import (FeatureSelector, IdMatcher,\n",
    "                                              featurestore_online_service)\n",
    "\n",
    "\n",
    "def serve_features(featurestore_name, entity_name, features, id):\n",
    "    feature_selector = FeatureSelector(id_matcher=IdMatcher(ids=features))\n",
    "\n",
    "    request = clients[\"feature_store_serving\"].read_feature_values(\n",
    "        featurestore_online_service.ReadFeatureValuesRequest(\n",
    "            # Fetch from the following feature store/entity type\n",
    "            entity_type=clients[\"feature_store\"].entity_type_path(\n",
    "                PROJECT_ID, REGION, featurestore_name, entity_name\n",
    "            ),\n",
    "            entity_id=id,\n",
    "            feature_selector=feature_selector,\n",
    "        )\n",
    "    )\n",
    "    return request\n",
    "\n",
    "\n",
    "features = serve_features(\n",
    "    FEATURESTORE_NAME, \"users\", [\"age\", \"gender\", \"liked_genres\"], \"alice\"\n",
    ")\n",
    "\n",
    "print(features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_serving:online,multi"
   },
   "source": [
    "### Multiple entity data items\n",
    "\n",
    "You serve features for multiple entity data items using the `streaming_read_feature_values()` method with the following parameters:\n",
    "\n",
    "- `entity_type`: The fully qualified resource identifier for the `EntityType` resource.\n",
    "- `feature_selector`: The features to serve from the corresponding `EntityType` resource.\n",
    "- `entity_ids`: The unique IDs of the data items to serve the corresponding features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "featurestore_serving:online,multi"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "header {\n",
      "  entity_type: \"projects/541313880005/locations/us-central1/featurestores/movies/entityTypes/users\"\n",
      "  feature_descriptors {\n",
      "    id: \"age\"\n",
      "  }\n",
      "  feature_descriptors {\n",
      "    id: \"gender\"\n",
      "  }\n",
      "  feature_descriptors {\n",
      "    id: \"liked_genres\"\n",
      "  }\n",
      "}\n",
      "\n",
      "entity_view {\n",
      "  entity_id: \"alice\"\n",
      "  data {\n",
      "    value {\n",
      "      int64_value: 55\n",
      "      metadata {\n",
      "        generate_time {\n",
      "          seconds: 1629493102\n",
      "          nanos: 261000000\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  data {\n",
      "    value {\n",
      "      string_value: \"Female\"\n",
      "      metadata {\n",
      "        generate_time {\n",
      "          seconds: 1629493102\n",
      "          nanos: 261000000\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  data {\n",
      "    value {\n",
      "      string_array_value {\n",
      "        values: \"Drama\"\n",
      "      }\n",
      "      metadata {\n",
      "        generate_time {\n",
      "          seconds: 1629493102\n",
      "          nanos: 261000000\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "\n",
      "entity_view {\n",
      "  entity_id: \"bob\"\n",
      "  data {\n",
      "    value {\n",
      "      int64_value: 35\n",
      "      metadata {\n",
      "        generate_time {\n",
      "          seconds: 1629493102\n",
      "          nanos: 261000000\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  data {\n",
      "    value {\n",
      "      string_value: \"Male\"\n",
      "      metadata {\n",
      "        generate_time {\n",
      "          seconds: 1629493102\n",
      "          nanos: 261000000\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  data {\n",
      "    value {\n",
      "      string_array_value {\n",
      "        values: \"Action\"\n",
      "        values: \"Adventure\"\n",
      "      }\n",
      "      metadata {\n",
      "        generate_time {\n",
      "          seconds: 1629493102\n",
      "          nanos: 261000000\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from google.cloud.aiplatform_v1.types import (FeatureSelector, IdMatcher,\n",
    "                                              featurestore_online_service)\n",
    "\n",
    "\n",
    "def serve_streaming_features(featurestore_name, entity_name, features, ids):\n",
    "    feature_selector = FeatureSelector(id_matcher=IdMatcher(ids=features))\n",
    "\n",
    "# TODO 3: Read Feature values for multiple entities\n",
    "    request = clients[\"feature_store_serving\"].streaming_read_feature_values(\n",
    "        featurestore_online_service.StreamingReadFeatureValuesRequest(\n",
    "            # Fetch from the following feature store/entity type\n",
    "            entity_type=clients[\"feature_store\"].entity_type_path(\n",
    "                PROJECT_ID, REGION, featurestore_name, entity_name\n",
    "            ),\n",
    "            entity_ids=ids,\n",
    "            feature_selector=feature_selector,\n",
    "        )\n",
    "    )\n",
    "    return request\n",
    "\n",
    "\n",
    "features = serve_streaming_features(\n",
    "    FEATURESTORE_NAME, \"users\", [\"age\", \"gender\", \"liked_genres\"], [\"alice\", \"bob\"]\n",
    ")\n",
    "\n",
    "for feature in features:\n",
    "    print(feature)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_serving:batch"
   },
   "source": [
    "## Batch Serving\n",
    "\n",
    "The Vertex AI Feature Store batch serving service is optimized for serving large batches of features in real-time with high-throughput, typically for training a model or batch prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_serving:batch,output,movies"
   },
   "source": [
    "### Output dataset\n",
    "\n",
    "For batch serving, you use a BigQuery table for the output. First, you must create this output destination table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "featurestore_serving:batch,output,movies"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created dataset: qwiklabs-gcp-04-ca29274815c5.movies_predictions_20220121182442\n"
     ]
    }
   ],
   "source": [
    "# Output dataset\n",
    "DESTINATION_DATASET = f\"movies_predictions_{TIMESTAMP}\"\n",
    "\n",
    "# Output table.\n",
    "DESTINATION_TABLE = \"training_data\"  # @param {type:\"string\"}\n",
    "\n",
    "DESTINATION_TABLE_URI = f\"bq://{PROJECT_ID}.{DESTINATION_DATASET}.{DESTINATION_TABLE}\"\n",
    "\n",
    "dataset_id = f\"{PROJECT_ID}.{DESTINATION_DATASET}\"\n",
    "dataset = bigquery.Dataset(dataset_id)\n",
    "dataset = bqclient.create_dataset(dataset)\n",
    "print(\"Created dataset:\", dataset_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_serving:batch,read,movies"
   },
   "source": [
    "### Batch Read Feature Values\n",
    "\n",
    "Assemble the request which specify the following info:\n",
    "\n",
    "*   Where is the label data, i.e., Table 1.\n",
    "*   Which features are read, i.e., the column names in Table 2.\n",
    "\n",
    "The output is stored in a BigQuery table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "featurestore_serving:batch,read,movies"
   },
   "outputs": [
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "request = featurestore_service.BatchReadFeatureValuesRequest(\n",
    "    # featurestore info\n",
    "    featurestore=clients[\"feature_store\"].featurestore_path(\n",
    "        PROJECT_ID, REGION, FEATURESTORE_NAME\n",
    "    ),\n",
    "    # URL for the label data, i.e., Table 1.\n",
    "    csv_read_instances=io.CsvSource(gcs_source=io.GcsSource(uris=[IMPORT_FILE])),\n",
    "    destination=featurestore_service.FeatureValueDestination(\n",
    "        bigquery_destination=io.BigQueryDestination(\n",
    "            # Output to BigQuery table created earlier\n",
    "            output_uri=DESTINATION_TABLE_URI\n",
    "        )\n",
    "    ),\n",
    "    entity_type_specs=[\n",
    "        featurestore_service.BatchReadFeatureValuesRequest.EntityTypeSpec(\n",
    "            # Read the 'age', 'gender' and 'liked_genres' features from the 'users' entity\n",
    "            entity_type_id=\"users\",\n",
    "            feature_selector=FeatureSelector(\n",
    "                id_matcher=IdMatcher(\n",
    "                    ids=[\n",
    "                        # features, use \"*\" if you want to select all features within this entity type\n",
    "                        \"age\",\n",
    "                        \"gender\",\n",
    "                        \"liked_genres\",\n",
    "                    ]\n",
    "                )\n",
    "            ),\n",
    "        ),\n",
    "        featurestore_service.BatchReadFeatureValuesRequest.EntityTypeSpec(\n",
    "            # TODO 4: Read the 'average_rating' and 'genres' feature values of the 'movies' entity\n",
    "            entity_type_id=\"movies\",\n",
    "            feature_selector=FeatureSelector(\n",
    "                id_matcher=IdMatcher(ids=[\"average_rating\", \"genres\"])\n",
    "            ),\n",
    "        ),\n",
    "    ],\n",
    ")\n",
    "\n",
    "response = clients[\"feature_store\"].batch_read_feature_values(request)\n",
    "\n",
    "response.result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "delete_bq_dataset"
   },
   "source": [
    "### Delete a BigQuery dataset\n",
    "\n",
    "Use the method `delete_dataset()` to delete a BigQuery dataset along with all its tables, by setting the parameter `delete_contents` to `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "delete_bq_dataset"
   },
   "outputs": [],
   "source": [
    "bqclient.delete_dataset(dataset, delete_contents=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "featurestore_delete"
   },
   "source": [
    "### Delete a `Featurestore` resource\n",
    "\n",
    "You can get a delete a specified `Featurestore` resource using the `delete_featurestores()` method, with the following parameters:\n",
    "\n",
    "- `name`: The fully qualified resource identifier for the `Featurestore` resource.\n",
    "- `force`: Forces deletion of the `Featurestore` resource when non-empty."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "featurestore_delete"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<google.api_core.operation.Operation at 0x7fa6dbf76510>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clients[\"feature_store\"].delete_featurestore(name=resource_name, force=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "name": "get_started_vertex_feature_store.ipynb",
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m87",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m87"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
